Hey @raboof, thanks for doing this! Let me get back to it in a few days, right now we're preparing RC.

Hey @raboof, do you have any idea what the possible implications of dropping the timestamps are?

Perhaps as alternative to zeroing them out we should introduce one or more parameters so that it becomes a client's decision whether to use currentTimeMillis/File.lastModified or not. WDYT?

@dwijnand I'm not aware of possible implications, and I didn't see any warnings on the reproducible builds maven plugin which also clears the timestamps.

In most cases these timestamps represent the 'time of last compilation' or 'time of packaging', I'm not sure when that'd be interesting information. Also I'm not sure this information is even exposed via the classpath API.

I'd be OK with making this behavior configurable (though of course I'd like the default to be 'cleared' ;) ). The question is how/where to configure this and how to pass that information around. I can give it a shot.

I still don't have time to look into this, but my gut feeling is that this could have an implication in Zinc to detect changes in jars as well as scalac (to avoid caching the classpath entries). Need to look deeper, but just throwing some observations out there...

Yeah, that's my concern too.

dbuild has checked the following projects against Scala 2.12:

❌ The result is: FAILED
(restart)

Aah, those kind of tools could be impacted, yes... that might be hard to deal with. Let's see if we can find an example.

The thing I'm most worried about is that today I learned that the JVM caches jars based on the timestamp (scala/bug#10295); which conflicts with this change... if a jar changes completely, and the JVM is caching the jars for the same compiler run, a reference to the previous jar could be maintained and that would cause compilation errors.

However, I'm not sure how all these pieces interact with each other. I need some kind of first hand experience because I can certainly ensure that will be the result.

@raboof I really appreciate you took this thing up. Would you mind creating a sbt plugin that overrides the sbt task creating a jar and nulls out the last modified time and experiment with it? I think that could give us great insights.

Interesting find! scala/bug#10295 (comment) is talking about java.util.File though - might that refer to the filesystem timestamp rather than the timestamp inside of the Jar (which this PR is about)?

@jvican It looks like IO.jar() is only called from the Package task, but I'm not sure it'd be possible/easy to override that task... could you point me in the right direction?

Right, so this only affects the files inside the zip files. Then I'm less scared. But I'm still positive that testing it should give us more confidence 😄 @raboof.

To override this in sbt, have a look at the implementation of packageBin, which is the responsible for packaging jars IIRC.

Let's check that we have project A, and project B, which depends on A. To check that this patch doesn't do anything dangerous (like affecting incremental compilation), I would override packageBin with the sbt syntax and reuse the return type, which is the jar (zip) file. It would be something similar to:

// remember to scope it in A
packageBin := {
  val packaged = packageBin.value
  // Write any timestamps inside the `packaged` file to zero
  // Do more stuff to make the jar reproducible
  ???
}

Following a similar methodology, you can instrument any task of sbt (which is one of the best features of the tool). If it's not by reusing the return type, it's by copy-pasting the original implementation defined in Defaults.scala and then modifying the bits you're interested in.

Let me know how it goes.

Hmm https://github.com/raboof/scala-reproducible-build-sbt-library-dependency doesn't appear to work even with the timestamps intact 🤔

What do you mean by "doesn't appear to work" @raboof?

Sorry that was not too helpful :).

When I sbt publishLocal test-lib and then start an interactive sbt session in project-using-test-lib and run, that works. When I then change the code in test-lib and sbt publishLocal it, runing the project-using-test-lib again from the (still running) interactive session I get [error] error while loading Constants, class file '/home/aengelen/.ivy2/local/net.bzzt/test-lib_2.12/0.1-SNAPSHOT/jars/test-lib_2.12.jar(net/bzzt/Constants.class)' is broken.

Did we expect this to work? If we didn't expect that to work, can you help come up with examples of things we do expect to work (and want to test it keeps working when we strip timestamps)?

But you're using exportJars right? I was not expecting that to work. What about generating the jars with packageBin and then having a dependency on that? What you want to check is that a change produces a different jar (with the same filestamp) but this jar is detected by Zinc and recompilation is triggered.

Note: The error you're seeing is because Scalac caches jars when run from sbt or in resident mode...

Here's a related Maven plugin that might serve as inspiration.

Right: there are basically 2 ways to make sure there are no in-jar timestamps in results: either not generating the timestamps in the first place, or post-processing the jars to remove the timestamps.

This PR works towards the former, while indeed the maven plugin and the sbt plugin I created on top of that post-process the jars to achieve the same.

For now the post-processing 'scratches my itch', but it would be nicer to solve this 'closer to the source'.

I'm afraid I'd need more hand-holding to get this over the finish line, so closing this for now. If anyone wants to reopen/continue, feel free (and I'd be happy to help), but I don't feel qualified to push it forward on my own right now :).

(continued in sbt/sbt#5233 and #279)